Lane recognition image processing apparatus

ABSTRACT

A lane recognition image processing apparatus can improve lane marking recognition performance by preventing false detection by addition of a condition without changing the basic principle of an one-dimensional image filter. A search area is set for each lane marking with respect to images stored in a image storage part through a window. A candidate point extraction part extracts candidate points of each lane marking from the search area thus set. A lane marking mathematical model equation is derived by approximating sets of extracted candidate points by a mathematical model equation. The candidate point extraction part includes a kernel size setting part, a filtering part that outputs, as a filtering result, the smaller of differences between the gray value of a pixel of interest and those of pixels forwardly and rearwardly apart a kernel size from the pixel of interest in a scanning direction, respectively, and a binarization part that binarizes the filtering result.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a lane recognition image processingapparatus which is installed on a vehicle for recognizing a lane of aroad based on the sensed or picked-up image of lane markings on the roadon which the vehicle is travelling, and which is applied to an advancedvehicle control system such as a lane departure warning system (LDWS)intended for use with preventive safety of the vehicle such as anautomobile, a lane keeping system (LKS) serving the purpose of reducinga cognitive load on drivers, etc. More particularly, the inventionrelates to a technique capable of improving reliability in the result ofthe recognition by providing vehicle lateral or transverse positioninformation in the lane.

2. Description of the Related Art

As a conventional lane recognition image processing apparatus, there hasbeen known one using an image filter (for example, see a first patentdocument: Japanese patent application laid-open No. H10-320549 (JP,H10-320549, A)).

This type of image filter is constructed of a relatively simple circuitthat can extract an area of a gray scale picture or image which isbrighter than its surroundings and which is less than or equal to apredetermined width.

The processing disclosed in the above-mentioned first patent document iscalled a one-dimensional image filtering process in which the gray valueg(h) of a pixel of interest is compared with the gray values g (h−Δh)and g (h+Δh) of pixels distant a kernel size Δh from the pixel ofinterest forwardly and rearwardly in a search scanning direction, andthe smaller value of the differences {g (h)−g (h−Δh)} and {g (h)−g(h+Δh)} thus obtained is made to be a filter output value.

In the conventional lane recognition image processing apparatus, whenthe forward view of the vehicle is taken by a camera installed thereonin a direction in which the vehicle is travelling, objects on an imagethus taken become linearly smaller toward a vanishing point. Therefore,when the width of the neighborhood of the pixel of interest (i.e., akernel size Δh) to be referenced or viewed by a one-dimensional imagefilter is fixed, the actual width of an area extracted by the filterincreases linearly in accordance with the increasing distance thereoffrom the camera. Accordingly, in case where a lane marking of apredetermined width on a road is detected as a physical quantity, thepossibility of the presence of objects other than the lane markingbecomes higher as the distance from the camera increases, so therearises a problem that reliability in the result of the recognition of adistant portion of the lane marking, which is needed to exactly graspthe shape of the road, is reduced.

In addition, when the image filter for use with the extraction of lanemarkings is applied to an road image that includes noise components ofhigh intensity in a range on the road, there will be another possibilityof misdetecting noise portions as lane markings. In particular, in casewhere a binarization threshold is controlled to decrease so as toextract degraded or thinned lane markings, or where a search areaincludes only high-intensity noise components but no lane marking suchas in the case of discontinuous portions of an intermittent lanemarking, there will be a problem that noise can be misdetected with avery high possibility.

Moreover, in the case of using a CMOS image sensor as an image sensingmeans, the CMOS image sensor is superior to a CCD image sensor withrespect to the reduction in size and cost of peripheral circuits, buthas a lower S/N ratio, so there is a higher possibility that the imagestaken by the CMOS image sensor contain noise. Accordingly, when thebinarization threshold of the image filter is controlled as usual withrespect to the images taken by the CMOS image sensor, the noisecomponent passes through the filter, thus giving rise to a problem ofdecreasing lane marking recognition performance

Further, in recent years, CMOS image sensors with a wide dynamic rangeare being developed, and intermittent high intensity parts are becomingvisually recognizable. However, when an image made to have a widedynamic range is expressed as a gray scale image of a plurality ofgradations (for instance, 256 steps), the entire image becomes a lowcontrast, so there arises a problem that in the ordinary control of thebinarization threshold, it is often difficult to extract lane markings.

Furthermore, in the conventional lane recognition image processingapparatus, lane markings are extracted by using one binarizationthreshold with respect to one image. Thus, in general, the contrast inthe output result of the image filter is high in near regions and low indistance regions, so there is a problem that in the case of extractinglane markings by the use of a single binarization threshold, it isimpossible to extract a distance lane marking though a near lane markingcan be extracted.

In addition, even if the binarization threshold is simply controlled todecrease in accordance with the increasing distance, there will happen asituation where the contrast can be varied at a distant or near locationdue to the shades of road structures depending upon the road-surroundingenvironment.

Moreover, in setting a window, in order to set the position of thewindow at a location including a lane marking and properly limit thesize of the window, it is appropriate to set a current window based onthe last window position calculated from a lane marking mathematicalmodel equation, but in a situation where the number of extractedcandidate points is limited and a lane marking mathematical modelequation cannot be derived (i.e., the state of lane markings being lostsight of), there exists no setting reference position, so it isnecessary to set a window of a wide or large size so as to search for alane marking from the entire screen. At this time, an extended period oftime for processing is required due to a wide or large search area.Therefore, it takes time for the condition to return from a lane markinglost-sight state to a lane marking recognition state, thus posing aproblem that the performance of the lane recognition image processingapparatus is reduced to a substantial extent.

Further, in lane recognition image processing, it has been proposed toextract top-hat shapes (i.e., having a constant width and a luminancehigher than that of the road surface) by using a one-dimensional imagefilter. However, such a proposal has a problem in that with respect toimages of low contrast or images of low S/N ratios taken by an imagesensor of a wide dynamic range, there is a possibility of misdetectingobjects other than lane markings, and that once a lane marking is lostsight of, it takes time until recognition of the lane marking isrestored.

SUMMARY OF THE INVENTION

An object of the present invention is to obtain a lane recognition imageprocessing apparatus which can be improved in lane marking recognitionperformance with reduced misdetection by the addition of a certaincondition, by variably setting the near luminance reference position andthe binarization threshold of a one-dimensional image filter inaccordance with the forward distance of an object in an image.

Another object of the present invention is to obtain a lane recognitionimage processing apparatus in which the binarization threshold has itslower limit set in accordance with the S/N ratio of the image to bebinarized so as to reduce misdetection resulting from an excessivedecrease in the binarization threshold, and in which the time ofrestoration from the lane marking lost-sight state can be shortened bysetting window-setting positions on a lane marking in a reliable mannerby sequentially setting of search area setting windows from a near sideto a remote or distant side so as to set the following window positionbased on the last extraction result, and at the same time by limitingthe window size.

Bearing the above objects in mind, according to the present invention,there is provided a lane recognition image processing apparatusinstalled on a vehicle for recognizing a lane based on a sensed image ofat least one lane marking on the surface of a road. The apparatusincludes: an image sensing part for sensing a forward view of thevehicle; an image storage part for temporarily storing images obtainedby the image sensing part; a window setting part for setting a searcharea for the at least one lane marking with respect to the images storedin the image storage part by means of a window; a candidate pointextraction part for extracting candidate points for the at least onelane marking from the search area set by the window setting part; and alane recognition part for deriving a lane marking mathematical modelequation by approximating sets of candidate points extracted by thecandidate point extraction part by a mathematical model equation. Thecandidate point extraction part includes: a kernel size setting partthat sets a kernel size Δh in accordance with a forward distance fromthe vehicle; a filtering part that outputs, as a filtering result, thesmaller one of the values that are obtained by two equations{g(h)−g(h−Δh)} and {g(h)−g(h+Δh)} using the gray value g(h) of a pixelof interest and the gray values g(h−Δh), g(h+Δh) of pixels forwardly andrearwardly apart the kernel size Δh from the pixel of interest in ascanning direction, respectively; and a binarization part that binarizesthe filtering result with a threshold.

According to the present invention, false detection or misdetection canbe reduced by adding a certain condition without changing the basicprinciple of the top-hat one-dimensional image filter, as a result ofwhich the lane marking recognition performance of the apparatus can beimproved to a substantial extent.

The above and other objects, features and advantages of the presentinvention will become more readily apparent to those skilled in the artfrom the following detailed description of preferred embodiments of thepresent invention taken in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing the schematic construction of a lanerecognition image processing apparatus according to a first embodimentof the present invention.

FIG. 2 is an external view showing a vehicle installing thereon the lanerecognition image processing apparatus according to the first embodimentof the present invention.

FIG. 3 is an explanatory view showing one example of a forward imageoutput from a camera in FIG. 2.

FIG. 4 is an explanatory view showing candidate points of individuallane markings in the forward image of FIG. 3.

FIG. 5 is an explanatory view showing a plurality of candidate points inthe forward image of FIG. 3.

FIG. 6 is an explanatory view showing two quadratic curves (lane markingmathematical model equations) each approximating a set of candidatepoints in FIG. 5.

FIG. 7 is an explanatory view showing the result of a filtering processcarried out on an original image luminance distribution according to thefirst embodiment of the present invention.

FIG. 8 is a flow chart showing filtering processing according to thefirst embodiment of the present invention.

FIG. 9 is an explanatory view showing a candidate point detectionprocess carried out by a binarization part according to the firstembodiment of the present invention.

FIG. 10 is an explanatory view showing the processing of a kernel sizesetting part according to the first embodiment of the present invention.

FIG. 11 is an explanatory view showing the results of near and distantfiltering processes according to the first embodiment of the presentinvention.

FIG. 12 is an explanatory view showing thresholds set with respect tothe results of near and distant filtering processes according to thefirst embodiment of the present invention.

FIG. 13 is an explanatory view showing a process of determining noiseranges and signal ranges with respect to the result of a filteringprocess according to the first embodiment of the present invention.

FIG. 14 is an explanatory view showing an intermittent lane marking.

FIG. 15 is a block diagram showing the schematic construction of a lanerecognition image processing apparatus according to a second embodimentof the present invention.

FIG. 16 is an explanatory view showing a window setting process based ontwo near candidate points according to the second embodiment of thepresent invention.

FIG. 17 is an explanatory view showing a window setting process based ona near candidate point and a vanishing point according to the secondembodiment of the present invention.

FIG. 18 is an explanatory view showing a vanishing point learningprocess according to the second embodiment of the present invention.

FIG. 19 is a flow chart showing window setting processing according tothe second embodiment of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Now, preferred embodiments of the present invention will be described indetail while referring to the accompanying drawings.

Embodiment 1

FIG. 1 is a block diagram that shows a lane recognition image processingapparatus according to a first embodiment of the present invention,wherein respective components thereof are illustrated so as tocorrespond to processing procedures. FIG. 2 is an external view thatillustrates a vehicle 2 on which a lane recognition image processingapparatus according to the first embodiment of the present invention isinstalled.

In FIG. 2, a camera 1, which constitutes an image sensing part, isinstalled on a front upper portion of the vehicle 2, and takes a forwardview of the vehicle 2.

In FIG. 1, the lane recognition image processing apparatus includes animage sensing part 101 having the camera 1 and installed on the vehicle2 see FIG. 2) for recognizing a lane of a road based on the picked-up orsensed images of lane markings on a road surface, an image storage part102 for temporarily storing the images obtained by the image sensingpart 101, a window setting part 103 for setting a search area of thelane markings with respect to the images stored in the image storagepart 102 through a window W, a candidate point extraction part 104 forextracting candidate points of a lane marking from the search area setby the window setting part 103, and a lane recognition part 105 forderiving a lane marking mathematical model equation by approximatingsets of candidate points extracted by the candidate point extractionpart 104 by a mathematical model equation.

The window setting part 103 includes a model equation reference part(not shown) and serves to set a reference position of the window W fromthe lane marking mathematical model equation.

The candidate point extraction part 104 includes a one-dimensional imagefiltering part 141 with a kernel size setting part 141 a, a binarizationpart 142 for binarizing the filtering results E of the one-dimensionalimage filtering part 141 by means of thresholds.

The kernel size setting part 141 a sets a kernel size Δh in accordancewith a forward distance from the vehicle 2.

The one-dimensional image filtering part 141 is constituted by a top-hatfilter, and outputs, as a filtering result E, the smaller one of thevalues that are obtained by equations {g(h)−g(h−Δh)} and {g(h)−g(h+Δh)}using the gray value g(h) of a pixel of interest and the gray valuesg(h−Δh), g(h+Δh) of distant pixels forwardly and rearwardly apart thekernel size Δh from the pixel of interest in a scanning direction,respectively.

The binarization part 142 includes a multi-threshold setting part 142 afor setting a threshold (described later) for each of search scanninglines of the one-dimensional image filtering part 141, an S/N ratiocalculation part 142 b for calculating an S/N ratio Rs of each filteringresult E, and a threshold lower limit setting part 142 c for setting alower limit for the thresholds based on the S/N ratio Rs.

The S/N ratio calculation part 142 b counts the number of filter passranges having their range widths less than a specified value as thenumber of noise ranges Mn in the filtering result E, also counts thenumber of filter pass ranges having their range widths more than orequal to the specified value as the number of signal ranges Ms in thefiltering result E, and calculates the S/N ratio Rs based on the numberof noise ranges Mn and the number of signal ranges Ms thus obtained.

The basic hardware configuration of the lane recognition imageprocessing apparatus shown in FIG. 1 is common to that of theconventional apparatus, but includes, as its concrete or detailedprocessing contents, the kernel size setting part 141 a, themulti-threshold setting part 142 a, the S/N ratio calculation part 142b, and the threshold lower limit setting part 142 c.

Now, a concrete processing operation of the lane recognition imageprocessing apparatus according to the first embodiment of the presentinvention as illustrated in FIGS. 1 and 2 will be described whilereferring to FIG. 3 through FIG. 6.

FIG. 3 is an explanatory view that shows one example of a forward imageoutput from the camera 1 that takes a picture of a forward road portionahead of the vehicle 2, wherein the state of right and left lanemarkings 3, 4 being taken a picture is illustrated.

FIG. 4 is an explanatory view that shows candidate points P1, P2 of thelane markings 3, 4, respectively, in a vehicle-forward image, wherein afiltering result E (an output of the top-hat filter) on a lane markingsearch line (hereinafter referred to simply as a “search line”) Vn, apair of right and left windows W1, W2 on the search line Vn, and thecandidate points P1, P2 are illustrated as being associated with oneanother.

FIG. 5 is an explanatory view that shows a plurality of candidate pointsP1, P2 in the forward image, illustrating an example in which sets ofcandidate points P1, P2 on a plurality of search lines Vn (n=0, 1, . . ., N−1) are detected along the lane markings 3, 4. In addition, in FIG.5, an arrow H indicates a horizontal direction, and an arrow V indicatesa vertical direction.

FIG. 6 is an explanatory view that shows two quadratic curves (lanemarking mathematical model equations) 7, 8 approximating sets ofcandidate points P1, P2, respectively, in FIG. 5, wherein the quadraticcurves 7, 8 approximated along the right and left lane markings 3, 4,respectively, are illustrated as being overlapped on the forward image.

First of all, the image sensing part 101 comprising the camera 1installed on the vehicle 2 takes a forward view of the vehicle 2, andacquires a sensed or picked-up image (see FIG. 3). At this time, let usassume that the right and left lane markings 3, 4 of the lane on whichthe vehicle 2 is traveling are fitted in the horizontal angle of view.

Here, it is also assumed that the image sensor installed on the camera 1comprises a CMOS image sensor.

The image storage part 102 takes the sensed or picked-up image of FIG. 3into the memory.

Subsequently, the window setting part 103 sets a pair of horizontalscanning ranges to search for the right and left candidate points P1, P2on a search line Vn (N=0, 1, . . . , N−1) constituting part of thesearch scanning lines as shown in FIG. 4 by means of the right and leftwindows W1, W2 (see broken line frames or boxes).

The right and left windows W1, W2 are set with the positions, which werecalculated by the lane marking mathematical model equations in the lastimage processing, being taken as setting reference positions.

In addition, the size of each of the windows W1, W2 is set according tothe maximum amount of movement of the lane markings 3, 4 generated in atime difference between the last image and the current image in theimages that are subject to the image processing. That is, the longer theperiod of image processing, the greater do the sizes of the windows W1,W2 become.

Next, the candidate point extraction part 104 scans the windows W1, W2(see FIG. 4) in the horizontal direction, performs filter processing bymeans of the one-dimensional image filtering part 141 with respect to anoriginal image luminance distribution D read from the image storage part(memory) 102, produces a filtering result E, and inputs it into thebinarization part 142.

At this time, by making reference to the gray value g(h) of a pixel ofinterest and the gray values g(h−Δh), g(h+Δh) of distant pixelsforwardly and rearwardly apart the kernel size Δh from the pixel ofinterest in a scanning direction, respectively, the one-dimensionalimage filtering part (top-hat filter) 141 outputs, as a filtering resultE, the smaller one of a difference {g(h)−g(h−Δh)} between the gray valueof the pixel of interest and that of the forward pixel and a difference{g(h)−g(h+Δh)} between the gray value of the pixel of interest and thatof the rearward pixel.

Moreover, the kernel size setting part 141 a sets the kernel size Δh inaccordance with the forward distance from the vehicle.

The binarization part 142 binarizes the filtering result E thus obtainedby the one-dimensional image filtering part 141 with the threshold,detects the candidate points P1, P2 and inputs them to the lanerecognition part 105.

The above-mentioned series of processes are carried out with respect toN search lines V0 through VN−1 (see FIG. 5), so that a correspondingnumber of sets of candidate points P1, P2 along the lane markings 3, 4are acquired.

Finally, the lane recognition part 105 acquires lane markingmathematical model equations 7, 8 (see FIG. 6) representative of thepertinent lane by approximating sets of the candidate points P1, P2 ofthe lane markings 3, 4 by means of appropriate mathematical modelequations (e.g., quadratic equations).

Thereafter, the processing operation of the lane recognition imageprocessing apparatus shown in FIG. 1 is terminated.

Next, the detailed processing operation of the one-dimensional imagefiltering part 141 will be described while referring to FIG. 7 throughFIG. 10.

Here, as stated before, a one-dimensional top-hat filter (hereinafterabbreviated as “T-H filter”) is used as the one-dimensional imagefiltering part 141.

FIG. 7 is an explanatory view that shows the filtering result E (T-Hfilter output) of the original image luminance distribution D, whereinthe axis of abscissa represents horizontal coordinates and the axis ofordinate represents intensity or luminance values (0–255).

In FIG. 7, there are illustrated luminance differences Δg1, Δg2 betweena point Po of interest and forward and rearward reference points Pa, Pbapart the kernel size Δh therefrom, respectively, on the original imageluminance distribution D.

Here, assuming that the individual luminance values of the point Po ofinterest and the reference points Pa, Pb are gPo, gPa and gPb, therespective luminance differences Δg1, Δg2 are respectively representedas follows.Δg1=gPo−gPaΔg2=gPo−gPb

The luminance value gPo of the point Po of interest corresponds to thegray value g(h) of a pixel of interest, and the luminance values gPa,gPb of the reference points Pa, Pb correspond to the gray valuesg(h−Δh), g(h+Δh) of the distant pixels forwardly and rearwardly apartthe kernel size Δh from the pixel of interest in the sear scanningdirection.

FIG. 8 is a flow chart that shows the T-H filter processing of theone-dimensional image filtering part 141.

FIG. 9 is an explanatory view that shows a process of detecting thecandidate points P1, P2, illustrating the state of a filtering result E(T-H filter output result) being binarized based on the threshold.

FIG. 10 is an explanatory view that shows the processing of the kernelsize setting part 141 a, illustrating the state of the T-H filteringkernel size Δh being variably set in accordance with the forwarddistance.

First of all, the one-dimensional image filtering part 141 sets thepoint Po of interest and the reference points Pa, Pb with respect to theoriginal image luminance distribution D that represents brightness by256 steps (luminance values 0–255), as shown in FIG. 7.

Here, the distance between the point Po of interest and the referencepoint Pa and the distance between the point Po of interest and thereference point Pb are respectively called the kernel size Δh, which isset in accordance with the forward distance by the kernel size settingpart 141 a, as shown in FIG. 10. The width of a filter pass range (to bedescribed later) is set by the kernel size Δh.

Specifically, the kernel size setting part 141 a individually sets thekernel size Δh for each search line Vn (i.e., in accordance with theforward distance), as shown in FIG. 10.

Accordingly, the kernel size Δh is set to be constant regardless of theforward distance when viewed from above.

Such a setting process for the kernel size Δh makes use of the fact thata sensed object is becoming linearly smaller toward a vanishing pointPz. Accordingly, if the nearest kernel size Δh is set to a widthcorresponding to the width of the lane markings 3, 4, the kernel size Δhon each search line Vn can be sequentially calculated by a linearinterpolation in accordance with the forward distance.

First of all, in the one-dimensional T-H filtering process as shown inFIG. 8, the luminance difference Δg1 between the point Po of interestand the reference point Pa and the luminance difference Δg2 between thepoint Po of interest and the reference point Pb, being set in a manneras shown in FIG. 7, are compared with each other, so that it isdetermined whether the relation of Δg1<Δg2 is satisfied (step S10).

When determined as Δg1<Δg2 in step S10 (i.e., Yes), it is subsequentlydetermined whether the luminance difference Δg1 (=gPo−gPa) is a positivevalue (step S11).

On the other hand, when determined as Δg1≧Δg2 in step S10 (i.e., No), itis subsequently determined whether the luminance difference Δg2(=gPo−gPb) is a positive value (step S12).

When determined as Δg1>0 in step S11 (i.e., Yes), the luminancedifference Δg1 is output as a filtering result E (T-H filter outputvalue) (step S13) and the processing routine of FIG. 8 is terminated.

When determined as Δg2>0 in step S12 (i.e., Yes), the luminancedifference Δg2 is output as a filtering result E (step S14), and theprocessing routine of FIG. 8 is terminated.

On the other hand, when determined as Δg1≦0 in step S11 (i.e., No), ordetermined as Δg2≦0 in step S12 (i.e., No), the filtering result E isset to “0” (step S15), and the processing routine of FIG. 8 isterminated.

Thus, the smaller value of the luminance differences Δg1, Δg2 isselected and output as a filtering result E.

For instance, in the case of the original image luminance distribution Dshown in FIG. 7, the relation of the luminance differences is Δg1<Δg2,so the control flow proceeds from step S10 to step S11, and if Δg1>0,the control flow proceeds to step S13 where the luminance difference Δg1becomes a filtering result E (T-H filter output value).

Here, note that if Δg1≦0, the control flow proceeds to step S15 wherethe filtering result E (T-H filter output value) becomes “0”.

Further, when the control flow has proceeded from step S10 to step S12(the luminance difference Δg2 has been selected), it is determinedwhether the luminance difference Δg2 is positive or negative, and thefiltering result E (T-H filter output value) is determined in step S14or step S15.

The above-mentioned series of processes in steps S10 through S15 areexecuted with respect to a point Po of interest within a window for eachsearch line Vn so that, as shown in FIG. 9, the filtering result E (seebroken line) with respect to the original image luminance distribution Dis obtained.

In FIG. 9, however, to simplify the explanation, only the filteringresult of a single line is illustrated.

Hereinafter, the binarization part 142 binarizes the filtering result Eso as to obtain the candidate points P1, P2 by using a binarization (T-Hfilter) threshold Th (hereinafter referred to simply as a “threshold”).

The threshold Th is set with respect to the filtering result E, as shownin FIG. 9, and serves to contribute to the detection of the right andleft candidate points P1, P2.

Here, note that though both of the positions of the right and leftcandidate points P1, P2 with respect to the areas extracted by thethreshold Th have been set within the corresponding lane areas,respectively, in FIG. 9, they may be set to a midpoint of the areaextracted by the threshold Th.

Here, reference will be made to a process of setting the threshold Th bymeans of the multi-threshold setting part 142 a in the binarization part142 while referring to FIG. 11 and FIG. 12.

FIG. 11 is an explanatory view that shows the results of near anddistant filterings (T-H filter output results), illustrating the statethat the contrast of the distant filtering result Eb is lower than thatof the near filtering result Ea.

FIG. 12 is an explanatory view that shows a threshold set with respectto the results of the near and distant filterings, illustrating thestate that both of the near and distant candidate points can be detectedby applying independent thresholds Tha, Thb to the results of the nearand distant filterings Ea, Eb, respectively.

The multi-threshold setting part 142 a in the binarization part 142individually sets a threshold Th for each search line Vn, similar to thesetting of the kernel size Δh (see FIG. 10).

For instance, it is assumed that the results of the near and distantfilterings Ea, Eb are obtained on near and distant search lines Va, Vb,respectively, as shown in FIG. 11.

Here, note that when attention is focused on the near search line Va, amaximum value Ea(max) and an average value Ea(mean) for the thresholdTha are calculated from the result of the near filtering Ea, and thethreshold Tha is set based on these values Ea(max), Ea(mean) as shown inthe following expression (1).Tha=Ea(max)−Ea(mean)  (1)

Also, the threshold Thb for the distant search line Vb is set in thesame manner. Hereinafter, an independent threshold Th for each searchline Vn is set in the same way.

As a consequence, a proper threshold Thb (<Tha) is set for the result ofthe distant filtering Eb, as shown in FIG. 12, whose contrast is lowerthan that in the result of the near filtering Ea.

Thus, by setting the luminance reference positions (reference points Pa,Pb) and the threshold Th for the filtering result E independently oneach search line Vn (i.e., in accordance with the forward distance)based on the kernel size Δh, an area with its width more than or equalto a predetermined width can be passed through the filter as a signalrange irrespective of the forward distance, so it is possible to achieveimage filter processing effective to extract the lane markings 3, 4 eachhaving a predetermined width.

Accordingly, false detection can be reduced to improve recognitionperformance for the lane markings 3, 4 merely by adding theabove-mentioned conditions without changing the basic principle of theone-dimensional image (top-hat) filter processing part 141.

In particular, by reducing false detection in the result of distantfiltering Eb that becomes low contrast (see FIG. 11 and FIG. 12), it ispossible to improve reliability in the distant lane marking recognitionresult needed to grasp the road shape.

That is, in the multi-threshold setting part 142 a, by setting athreshold Th for each search line (search scanning line) Vn of theone-dimensional image filtering part 141, and by setting a properdistant threshold Thb (<Tha) with respect to a distant image whosecontrast is lower than that of a near image, as shown in FIG. 12, thedistant lane marking recognition performance can be improved.

By setting the threshold Th for each search line Vn, it is possible tocope with a situation where the near contrast (i.e., the contrast of anear location) is conversely lowered due to the shadow of a roadstructure, etc.

In addition, by sequentially setting a window W for each search line Vn(from a near side toward a remote or distant side) with the use of thelane marking mathematical model equation, and by setting the followingwindow position based on the last extraction result, it is possible toset the position of the window W on each of the lane markings 3, 4 in areliable manner. Moreover, by limiting the size of each window W, therestoration time from the lost-sight state of the lane markings 3, 4 canbe shortened.

Further, since the dynamic range of the processing operation in theone-dimensional image filtering part 141 and the binarization part 142is wide, the binarization threshold can be properly set with respect toan image which is taken by the use of a CMOS image sensor of a widedynamic range and in which the contrast of the entire image is low.

Now, reference will be made to an arithmetic process of calculating theS/N ratio Rs by means of the S/N ratio calculation part 142 b whilereferring to FIG. 13.

FIG. 13 is an explanatory view that shows a process of determiningsignal ranges and noise ranges with respect to the filtering result E,wherein the axis of abscissa represents horizontal coordinates and theaxis of ordinate represents the output levels of the filtering result E.

The S/N ratio calculation part 142 b detects noise ranges 20 togetherwith signal ranges, as shown in FIG. 13, calculates an S/N ratio Rs fromthe number of the signal ranges and the number of the noise ranges, andutilizes the S/N ratio Rs thus obtained as a setting condition for thethreshold Th.

In FIG. 13, first of all, the S/N ratio calculation part 142 b extractsfilter pass ranges (see shaded portions) by utilizing the threshold Thwith respect to the filtering result E.

Subsequently, the width d of each filter pass range is compared with aspecified value, and it is determined that those which have their rangewidth d greater than or equal to the specified value are the signalranges, and those which have their range width less than the specifiedvalue are the noise ranges 20.

In addition, the number Ms of the signal ranges and the number Mn of thenoise ranges 20 are counted, respectively, and the value calculatedaccording to the following expression (2) by using the number of thesignal ranges Ms and the number of the noise ranges Mn is defined as theS/N ratio Rs.Rs=Ms/(Ms+Mn)×100[%]  (2)

Next, reference will be made to a process of setting a lower limit ofthe threshold Th by means of the threshold lower limit setting part 142c.

The threshold lower limit setting part 142 c sets the lower limit of thethreshold Th based on the S/N ratio Rs calculated by the S/N ratiocalculation part 142 b. Specifically, the threshold Th is controlled soas to keep the S/N ratio Rs to be constant.

For instance, when the permissible lower limit value of the S/N ratio Rsis adjusted to 70%, thresholds Th(70%) when the S/N ratio Rs satisfies70% or more are always stored, and a control process of adopting thelatest threshold Th(70%) (stored at the last) is applied when the S/Nratio Rs has become less than 70%.

Thus, by setting the lower limit of the threshold Th based on the S/Nratio Rs of the image in the threshold lower limit setting part 142 c soas to reduce false detection that would otherwise result from anexcessive decrease or lowering of the threshold Th, it is possible togreatly reduce the false detection due to such an excessive lowering ofthe threshold Th with respect to images containing a lot of noise.

Particularly, in cases where no lane marking exists in the window W1when the vehicle is traveling on a lane with an intermittent lanemarking 3, as shown in FIG. 14, false detection can be effectivelyreduced.

Moreover, when a CMOS image sensor is used as the image sensing part101, the S/N ratio of an image sensed thereby decreases as compared withthe case of using a CCD image sensor. However, even if the CMOS imagesensor is used, it is possible to achieve substantially the samerecognition performance of the lane markings 3, 4 as in the case ofusing the CCD image sensor by setting the lower limit of the thresholdTh in accordance with the S/N ratio Rs.

Embodiment 2.

Although in the above-mentioned first embodiment, only the modelequation reference part is used in the setting of a window, a candidatepoint reference part 131 b and a vanishing point reference part 131 ccan be added to or incorporated in the reference position setting part131 in a window setting part 103A, and a vanishing point learning part106 can also be provided for optimally setting the window W for eachsearch line Vn, as shown in FIG. 15.

FIG. 15 is a block diagram showing a lane recognition image processingapparatus according to a second embodiment of the present invention in amanner to correspond to processing procedures, wherein the same orcorresponding parts or elements as those in the above-mentioned firstembodiment (see FIG. 1) are identified by the same symbols whileomitting a detailed description thereof.

In FIG. 15, a major difference from the above-mentioned first embodiment(FIG. 1) is that the reference position setting part 131 in the windowsetting part 103A incorporates therein not only the model equationreference part 131 a but also the candidate point reference part 131 band the vanishing point reference part 131 c, and at the same time,provision is made for the vanishing point learning part 106 inconjunction with the vanishing point reference part 131 c.

The reference position setting part 131 includes the model equationreference part 131 a, the candidate point reference part 131 b, and thevanishing point reference part 131 c, so that either one of the modelequation reference part 131 a, the candidate point reference part 131 band the vanishing point reference part 131 c can be selected to set thereference positions of windows W to search for the lane markings 3, 4.

The model equation reference part 131 a in the window setting part 103Aserves to set the reference positions of the windows W on each searchline Vn from the above-mentioned lane marking mathematical modelequations.

FIG. 16 is an explanatory view that shows the processing of thecandidate point reference part 131 b, wherein attention is expedientlyfocused on the left lane marking 3 alone so as to set a referenceposition with respect to a left window W1.

In FIG. 16, in cases where there exist two or more candidate points Pq,Pr, the candidate point reference part 131 b in the reference positionsetting part 131 sets the window W1 on the following search line Vnbased on a straight line Lqr connecting between the two adjacentcandidate points Pq, Pr. That is, the window W1 is set based on anintersection Px between the straight line Lqr and the search line Vn.

FIG. 17 is an explanatory view that shows the processing of thevanishing point reference part 131 c, wherein similar to the case ofFIG. 16, attention is focused on the left lane marking 3 alone so as toset a reference position with respect to the left window W1.

In FIG. 17, in cases where there exists a single candidate point Pqalone, the vanishing point reference part 131 c in the window settingpart 103A sets the window W1 on the following search line Vn based on astraight line Lqz connecting between the near candidate point Pq and avanishing point Pz. That is, the window W1 is set based on anintersection Py between the straight line Lqz and the search line Vn.

FIG. 18 is an explanatory view that shows the processing of thevanishing point learning part 106, illustrating the state that thevanishing point (learning position) Pz is obtained through learningbased on the right and left lane markings 3, 4.

In FIG. 18, the vanishing point learning part 106 approximates sets ofright and left candidate points P1, P2 (see FIGS. 4, 5 and FIG. 9) inthe vicinity of the vehicle extracted by the candidate point extractionpart 104 by linear or straight lines, respectively, and learns as thevanishing point Pz an intersection between the approximate linear linesLz1, Lz2 (corresponding to the right and left lane markings 3, 4)derived from the sets of candidate points.

Now, reference will be made to a process of setting a window W by meansof the lane recognition image processing apparatus according to thesecond embodiment of the present invention shown in FIG. 15 whilereferring to a flow chart of FIG. 19 together with FIG. 16 through FIG.18.

In FIG. 19, steps S20 through S22 represent a determination process forselecting the reference parts 131 a through 131 c, respectively, andsteps S23 through S26 represent a process for setting a referenceposition of each window W based on the results of determinations in therespective steps S20 through S22.

The window setting part 103A first determines whether there exists anylane marking mathematical model equation (step S20), and when determinedthat a lane marking mathematical model equation exists (i.e., Yes), itthen selects the model equation reference part 131 a. That is, similarto the above, the position of the line Lqr on a search line Vn iscalculated from the lane marking mathematical model equation, and it isdecided as the reference position of the window W (step S23).

On the other hand, when determined in step S20 that there exists no lanemarking mathematical model equation (i.e., No), it is subsequentlydetermined whether two or more candidate points have been extracted(step S21).

When determined in step S21 that two or more candidate points have beenextracted (i.e., Yes), the candidate point reference part 131 b isselected, so that it decides an intersection Px between the straightline Lqr connecting the candidate point Pq and the candidate point Prand the following search line Vn as a reference position, as shown inFIG. 16 (step S24).

At this time, assuming that the lane marking 3 is searched in adirection from a near side toward a distance side, there exist thecandidate point Pq initially detected and the candidate point Pr nextdetected.

On the other hand, when determined in step S21 that two or morecandidate points have not been extracted (i.e., No), it is furtherdetermined whether a single candidate point alone has been extracted(step S22).

When determined in step S22 that a single candidate point alone has beenextracted (i.e., Yes), the vanishing point reference part 131 c isselected, so that it decides as a reference position an intersection Pybetween a straight line Lqz connecting the near candidate point Pq andthe vanishing point Pz and the following search line Vn, as shown inFIG. 17 (step S25).

In this case, too, assuming that a search is started from a near sidetoward a distance side, there exists the candidate point Pq initiallydetected.

On the other hand, when determined in step S22 that there is nocandidate point at all (i.e., No), a search is made for the lane marking3 from the entire image sensing screen (step S26).

Hereinafter, subsequent to the reference position setting steps S23through S26, a window W1 is set for the left lane marking 3 for instance(step S27). Though not described in detail, a window W2 is similarly setfor the right lane marking 4 according to the same process steps.

Then, candidate points P1, P2 are extracted by means of the windows W1,W2 set in step S27 (step S28), and it is determined whether the searchline Vn being currently processed is the final line (n=N−1)(step S29).

When determined in step S29 that the current search line Vn is the finalline (i.e., Yes), the processing routine of FIG. 19 is ended, whereaswhen determined that the current search line Vn is not the final line(i.e., No), a return is performed to step 20, from which theabove-mentioned processes are repeated until the final line is reached.

Here, note that the vanishing point learning part 106 obtains thelearning coordinates of the vanishing point Pz from the approximatestraight lines Lz1, Lz2 of the right and left lane markings 3, 4, asshown in FIG. 18, and inputs them to the vanishing point reference part131 c, thus contributing to the reference position setting process instep S25.

For instance, if there is a state in which a sufficient number ofcandidate points have been extracted so as to permit the acquisition ofthe approximate straight lines Lz1, Lz2 (see FIG. 18) before the singlecandidate point Pq alone comes into existence, the vanishing pointlearning part 106 provides the learning coordinates of the finalvanishing point Pz by low-pass filtering the coordinates of anintersection between the approximate straight lines Lz1, Lz2.

On the other hand, if there is no state in which the approximatestraight lines Lz1, Lz2 have been obtained before the single candidatepoint Pq alone comes into existence, a vanishing point default position(i.e., calculated from the mounting height and the angle of elevation ofthe camera 1 (see FIG. 2)) in the image sensing screen is substituted asthe vanishing point Pz.

Accordingly, in either case, the learning coordinates of the vanishingpoint Pz can be obtained in a reliable manner, and in step S25, theintersection Py (see FIG. 17) between the straight line Lqz connectingthe vanishing point Pz and the initially detected candidate point Pq andthe following search line Vn can be set as the reference position.

Further, the result of the process in the candidate point extractionstep S28 among a series of processes shown in FIG. 19 is reflected ineach of the determination steps S21 and S22. That is, if a search isexecuted normally, the number of candidate points extracted in step S28increases as the search proceeds from the near side toward the distantside. Accordingly, if focusing on the step S22 for instance, the resultthereof will be changed from the state of branching to step S26 into thestate of branching to step S25.

Similarly, when focusing on step S21, the result thereof will be changedfrom the state of branching to step S25 into the state of branching tostep S24.

However, when the curvature of the road is relatively large at the timeof using the vanishing point Pz and the candidate point Pq in step S25,the shape of the straight line Lqz connecting the vanishing point Pz andthe candidate point Pq and the shape of the lane marking 3 becomemismatch or disagreement with each other as the distance from thecandidate point Pq increases.

To cope with such a problem, the following measure can be taken. Thatis, assuming that the horizontal angle of visibility of the camera 1 is33 degrees and the mounting height thereof is 1.2 meter, for example, arange of 20 meter or less forward from the camera 1 can be considered asa straight line, and the execution condition for the process in step S25(i.e., the intersection Py between the straight line Lqz connecting thevanishing point Pz and the candidate point Pq and the search line Vn istaken as a search reference position) is limited to within a range of 20meter or less forward from the camera 1.

Thus, in the case of the presence of two or more candidate points, bysetting as a reference position the intersection Px between the straightline Lqr connecting the two candidate points Pq, Pr and the followingsearch line Vn, it is possible to set windows W on the lane markings 3,4, respectively, even in the state where the last lane markingmathematical model equations (window setting reference) are not present(i.e., the lost-sight state of the lane markings 3, 4).

At this time, if the two most distant possible candidate points aresequentially extracted as targets in the case of extracting candidatepoints from a near side toward a distant side in a sequential manner,the ability to follow the lane markings 3, 4 can be improved withrespect to a straight road as well as a road with a curvature.

In addition, the process of setting proper windows serves to prevent thewindows W from being set wider than necessary, so the processing timecan be shortened, and restoration from a lost-sight state of the lanemarkings 3, 4 to a recognition state thereof can be carried out in ashort time.

Moreover, by approximating sets of candidate points P1, P2 by straightlines, and by learning the vanishing point Pz from the intersectionbetween the straight lines Lz1, Lz2 that proximate the right and leftlane markings 3, 4, respectively, it is possible to set windows W on thelane markings 3, 4 on the basis of the intersection Py of the straightline Pqz connecting the candidate point Pq and the vanishing point Pzand the following search line Vn, even if the lane markings are lostsight of with the presence of the single candidate point alone.

In particular, even if the vehicle 2 (see FIG. 2) moves sideways on astraight road, the vanishing point Pz does not move as long as thevehicle 2 is traveling in parallel to the road, as a consequence ofwhich the windows W can be set on the lane markings 3, 4 in a reliablemanner.

Further, by sequentially setting a window W for each search line Vn(from a near side toward a distant side), and by setting the followingwindow position based on the last extraction result, it is possible toset the position of the window W on each of the lane markings 3, 4 in areliable manner. Furthermore, by limiting the size of each window W, therestoration time from the lost-sight state of the lane markings 3, 4 canbe shortened.

While the invention has been described in terms of preferredembodiments, those skilled in the art will recognize that the inventioncan be practiced with modifications within the spirit and scope of theappended claims.

1. A lane recognition image processing apparatus installed on a vehiclefor recognizing a lane based on a sensed image of at least one lanemarking on the surface of a road, said apparatus comprising: an imagesensing part sensing a forward view of said vehicle; an image storagepart temporarily storing images obtained by said image sensing part; awindow setting part setting a search area for said at least one lanemarking with respect to the images stored in said image storage part bymeans of a window; a candidate point extraction part extractingcandidate points for said at least one lane marking from said searcharea set by said window setting part; and a lane recognition partderiving a lane marking mathematical model equation by approximatingsets of candidate points extracted by said candidate point extractionpart by a mathematical model equation, wherein said candidate pointextraction part comprises: a kernel size setting part setting a kernelsize Δh in accordance with a forward distance from said vehicle; aone-dimensional image filtering part outputting, as a filtering result,the smaller one of the values that are obtained by two equations{g(h)−g(h−Δh)} and {g(h)−g(h+Δh)} using the gray value g(h) of a pixelof interest and the gray values g(h−Δh), g(h+Δh) of pixels forwardly andrearwardly apart said kernel size Δh from said pixel of interest in ascanning direction, respectively; and a binarization part that binarizessaid filtering result with a threshold.
 2. The lane recognition imageprocessing apparatus as set forth in claim 1, wherein said binarizationpart comprises: an S/N ratio calculation part calculating an S/N ratioof said filtering result; and a threshold lower limit setting partsetting a lower limit of said threshold based on said S/N ratio; whereinsaid S/N ratio calculation part counts the number of filter pass rangeshaving their range widths less than a specified value in said filteringresult as the number of noise ranges, also counts the number of filterpass ranges having their range widths greater than or equal to thespecified value in said filtering result as the number of signal ranges,and calculates said S/N ratio based on said number of signal ranges andsaid number of noise ranges.
 3. The lane recognition image processingapparatus as set forth in claim 2, wherein said image sensing partcomprises a CMOS image sensor.
 4. The lane recognition image processingapparatus as set forth in claim 1, wherein said binarization partincludes a multi-threshold setting part setting said threshold for eachsearch scanning line of said one-dimensional image filtering part. 5.The lane recognition image processing apparatus as set forth in claim 1,wherein said window setting part includes a candidate point referencepart setting, in the case of said candidate points being two or more, awindow based on an intersection between a straight line connecting twopoints among said candidate points and the following search scanningline.
 6. The lane recognition image processing apparatus as set forth inclaim 1, wherein said at least one lane marking comprises a pair ofright and left lane markings, and said apparatus further comprises avanishing point learning part obtaining a vanishing point learningposition based on said right and left lane markings; said vanishingpoint learning part approximates sets of candidate points extracted bysaid candidate point extraction part by straight lines, and obtains saidvanishing point learning position from an intersection betweenapproximate straight lines of said right and left lane markings; andsaid window setting part includes a vanishing point reference part thatsets said window on the basis of an intersection between a straight lineconnecting one of said candidate points and said vanishing pointlearning position and the following search scanning line.